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Abstract. As incremental Structure from Motion algorithms become 
effective, a good sparse point cloud representing the map of the scene 
becomes available frame-by-frame. From the 3D Delaunay triangulation 
of these points, state-of-the-art algorithms build a manifold rough model 
of the scene. These algorithms integrate incrementally new points to the 
3D reconstruction only if their position estimate does not change. Indeed, 
whenever a point moves in a 3D Delaunay triangulation, for instance 
because its estimation gets refined, a set of tetrahedra have to be removed 
and replaced with new ones to maintain the Delaunay property; the 
management of the manifold reconstruction becomes thus complex and 
it entails a potentially big overhead. In this paper we investigate different 
approaches and we propose an efficient policy to deal with moving points 
in the manifold estimation process. We tested our approach with four 
sequences of the KITTI dataset and we show the effectiveness of our 
proposal in comparison with state-of-the-art approaches. 


1 Introduction 

Incremental 3D reconstruction from a sparse point cloud is gaining interest in the 
computer vision community as incremental Structure from Motion algorithms 
are consolidating |18j . This is clearly true for those applications where a rough, 
but dense, surface represents a sufficient and effective representation of the scene, 
e.g, for traversability analysis in unmanned vehicle navigation. Furthermore, in 
real-time applications, the map of the environment needs to be updated online, 
and the surface has to be estimated incrementally. 

Most of the existing algorithms |10n3l8lh) bootstrap the reconstruction of a 
mesh surface from the 3D Delaunay triangulation of a sparse point cloud. Indeed, 
the 3D Delaunay triangulation is very powerful: the Delaunay property, i.e., no 
point of the triangulation is inside the sphere circumscribing any tetrahedron, 
avoids as much as possible the resulting tetrahedra to have a degenerate shape 
m ; it is self-adaptive, i.e., the more the points are dense the more the tetrahedra 
are small; it is very fast to compute, and to update against point removal or 
addition; off-the-shelf libraries, such as CGAL m, enable a very simple and 
efficient management of it. 

As soon as a Delaunay triangulation is available, several approaches exist to 
extract a surface taking into account the visibility of each point. The simplest 




algorithm is the Space Carving [^: it initializes all the tetrahedra as matter^ 
then it marks as free space the tetrahedra intersected by the camera-to-point 
viewing rays, i.e., the lines from the camera center to the observed 3D points in 
the triangulation. The boundary between free space and matter represents the 
final surface of the scene. Pan et al. m improve upon this simple procedure by 
proposing an online probabilistic Space Carving, but this is not an incremental 
approach: they start from scratch every time new points are added. Lovi et al. 
m present the first incremental Space Carving algorithm which runs real-time, 
but, as for the previous methods, the estimated surface is not guaranteed to be 
manifold 

Several reasons lead to enforce the manifold property as explained in [7]. 
Most Computer Graphics algorithms need the manifold property, for instance 
smoothing with Laplace-Beltrami operator [12] , or the linear mesh parametriza- 
tion m- Moreover the manifold property enables surface evolution in mesh- 
based Multi-View Stereo, as in mm .the manifold property enables a photomet¬ 
ric rehnement by surface evolution such as with the high accurate Multi-View 
Stereo mesh-based algorithm as in mm- With these approaches is hard to esti¬ 
mate the surface evolving flow in the presence of non manifold vertices: indeed 
they compute for each vertex the gradient minimizing the reprojection error, by 
summing-up the contribution of the incident facets; if the vertex is not manifold, 
this gradient does not converge. As a further proof of this, HU needs to manually 
fix the surface estimated via s-t cut. As in [m, it is possible to fix the mesh as a 
post-processing step, but reconstructing directly a manifold as in the proposed 
paper, enables the design of a fully automatic pipeline which do not need human 
intervention. 

In literature, the only algorithm reconstructing a manifold incrementally was 
proposed by Litvinov and Lhuiller |8I9) . In their work, the authors bootstrap from 
the Space Carving procedure and, by taking into account the number of inter¬ 
sections of each tetrahedron with the viewing rays, they reconstruct a surface 
keeping the manifold property valid. The main limitation is that Litvinov and 
Lhuiller insert a point into the Delaunay triangulation only when its position is 
definitive, then they cannot move the point position anymore even in the case 
they could refine their estimate. The main reason of Litvinov and Lhuiller design 
choice has to be ascribed to the computational cost of updating the visibility 
information along the viewing rays incident to each moved point, and the com¬ 
putational cost of updating part of the Delaunay triangulation, which in turn 
induces a new manifold reconstruction iteration step. 

Indeed, the very common approach to deal with a point moving in the trian¬ 
gulation, is to remove it and add it back in the new position [16] (Fig.[^. When 
we remove a point (the point A in Fig. [^a)) and we want to keep the Delaunay 
property, we have to remove all the tetrahedra incident to that point (light red 
triangles in Fig. [^b)); then, we add a new set of tetrahedra to triangulate the 
resulting hole (dark green triangles in[^c)). When we add a new point into the 
triangulation (the point B in Fig. [^d)), a set of tetrahedra would conflict with 
it, i.e., the Delaunay property is broken (light red triangles in Fig. [^d)); so. 





Fig. 1. Example of point removal in 2D case. Light red triangles depict are removed 
and replaced with the new dark green ones. 


we remove this set of tetrahedra again (red triangles in Fig. [^e)) and we add 
a new connected set that re-triangulate the hole (dark green triangles in Fig. 
0f))- Whenever a set of tetrahedra is replaced, we have to transfer conveniently 
the information about the visibility (matter or free space) of the removed tetra¬ 
hedra to the new one. In addition to this, we have to update the visibility of the 
tetrahedra crossed by a visibility ray from one camera to the moved point. For 
these reasons the update of the point position is computational demanding. 

To complete the overview of the incremental reconstruction methods from 
sparse data, we mention here another very different approach was proposed 
by Hoppe et al. who label the tetrahedra with a random field, and extract 
the surface via graph-cuts by minimizing a visibility-consistent energy function. 
This incremental algorithm is effective and handles the moving points, but the 
manifold property of the reconstructed surface is not yet guaranteed. 

In this paper we propose, to the best of our knowledge, the first manifold 
3D reconstruction algorithm from sparse data which deals with dynamic point 
changes. In particular, we show that in this setting the algorithm by Lovi et 
al. [10] provides a feasible solution, but it is very inefficient and we propose a 
novel efficient policy to handle the visibility update of Delaunay tetrahedra with 
moving points. 

In Section we summarize a slightly modified version of the approach of 
[S] we use to reconstruct a manifold surface. In Section we describe Lovi’s 
approach m and our proposal to deal with moving points, together with a 
complexity analysis that explains why our approach is more efficient. In Section 
[^we show the experimental results on the publicly available dataset KITTI [2, 
while, in Section]^ we point out some future works and in the conclusion of the 
paper. 

2 Manifold Reconstruction 

In this paper we reconstruct a manifold surface that represents the observed 
scene. A surface is manifold if and only if the neighborhood of each point is 
homeomorphic to a disk. In the discrete case, the points are the vertexes of a 
mesh, and the neighborhood is represented by the incident triangles (or poly¬ 
gons); a surface is manifold if each vertex v is regular, i.e., if and only if the 
edges opposite to v form a closed path without loops (see [8] for more details). 








2.1 Incremental manifold extraction with tetrahedra weighting 

In this section we briefly summarize our variation on the method originally 
proposed in [H] enhanced by a weighting scheme that avoids the creation of most 
visual artifact in the final mesh (more discussion about visual artifacts in 0 )- 
In our Space Carving algorithm, a weight roughly represents how many rays 
intersect a tetrahedron, and in the following, a tetrahedron belongs to free space 
if its weight w is higher than a threshold (in our case = 1.0). 

Sparse point cloud The input of our algorithm is a sparse 3D point cloud, 
estimated incrementally by assuming the camera poses to be known. For each 
keyframe, i.e., every K = 5 frames, we extract Edge-point features, i.e. 2D points 
laying on the image edges El; these points represent measurements of 3D points. 
Frame-by-frame we track these features with the Kanade-Lucas-Tomasi tracker, 
and, at each keyframe, we estimate the new positions of the 3D points with the 
new measurements available from the tracking. New estimates are obtained by 
triangulating the 2D tracked points and by minimizing the reprojection error 
with a Gauss-Newton algorithm. Once a new estimate of a 3D point is available, 
we add it to the Delaunay triangulation, i.e., to the reconstruction; then we 
update its position according to the new measurements: this update induces the 
motion of the points inside the triangulation. 


3D reconstruction The reconstruction of the surface bootstraps from the man¬ 
ifold partitioning the 3D triangulation between the set O of outside tetrahedra, 
i.e., the manifold subset of the free space (not all the free space tetrahedra will 
be part of the manifold), and the complementary set / of inside tetrahedra, i.e. 
the remaining tetrahedra that represent the matter together with the free space 
tetrahedra which would invalidate the manifold property. 

Let i5(Oii„it) be the initial manifold. This initial manifold is obtained with the 
following steps. Point Insertion: add all the 3D points estimated up to time finit 
and build thir 3D Delaunay triangulation. Ray tracing and tetrahedra weighting: 
for each viewing ray, the algorithm traverses the triangulation and adds a weight 
wi = I.O to the intersected tetrahedra, a weight = 0.8 to the neighbors and 
a weight W 2 = 0.2 to the neighbors of their neighbors. Such weighting scheme 
acts as a smoother of the visibility and avoids the creation of visual artifacts; 
it is the main difference between our algorithm and the algorithms proposed in 
|8I9| . Growing: initialize a queue Q starting from the tetrahedron with the higher 
weight. Then: (a) pick the tetrahedron with highest weight from Q and add it 
to Otinit only if the resulting surface between and remains manifold; 

(b) if inserted add the neighboring tetrahedra to the queue Q, otherwise discard 
it; continue iteratively until Q is empty. 

Once the system is initialized, a new set of points Pt^ is estimated at each 
tk = bnit P k *Tk frame, named keyframes, where k € N“'" and Tk is the inverse 
of the keyframe rate. The insertion of a point p € Pt^ causes the removal of the 
set of tetrahedra breaking the Delaunay property, and, the surface 5{Ot^) = 


\ Dt^.) is not guaranteed to be manifold anymore. To avoid this, the 
authors in [S] define a list of tetrahedra Et^. D Dt^. and apply the Shrinking 
procedure, i.e., the inverse of Growing: they subtract iteratively from Ot^_^ the 
tetrahedra A G Et^ keeping the manifoldness valid. After this process, it is 
likely that Dt^ n = 0. Whenever Dt^ n Ot^ 0 the point p is not added to 
the triangulation, i.e., is dropped. Once all points in Pt^ have been added (or 
dropped), the growing process runs similarly to the initialization procedure, but 
the queue Q is initialized with the tetrahedra A G T\0 such that A n 60 ^ 0. 

3 Reconstructing a manifold with moving points 

As previously described, Litvinov and Lhuiller [S] algorithm adds points to the 
triangulation only when their 3D position is completely defined; by doing this, 
there are no changes in the Delaunay triangulation, induced by moving points. 
This results in a restriction if we would like to refine the estimation of the position 
of a point 3D position after its insertion. 

Only Lovi et al. m presents an incremental Space Carving algorithm which 
deals with moving points, but their method does not enforce the manifold prop¬ 
erty. In this paper we verify the approach of Lovi et al. [lOj to be very inefficient 
for manifold reconstruction, and we present a different approach to deal with 
moving points that leads to a significant faster computation. 

3.1 The straightforward approach 

The simplest way to deal with moving points while reconstructing a manifold 
surface, is to apply a straightforward modification to the so called Refinement 
Event Handler by Lovi et al. in m- The Refinement Event Handler algorithm 
assumes that, for each tetrahedron in the Delaunay triangulation a list of the 
intersecting viewing rays is stored. In our voting schema an intersecting ray is 
each ray that increase the weight of the tetrahedron. 

Let poid be a point that moves to position Pnew, the algorithm in uni moves 
the point by removing point poU and adding Pnew as a new point, according to 
the classical approach of [2] , then for each point they apply the following steps. 
Rays collection: collect in a set U all the rays stored into the tetrahedra incident 
to Poid) i-O., those affected by the Poid removal (e.g., the light red triangle in Fig. 
[^a)). Vertex removal: remove the vertex poid and its neighboring tetrahedra 
from the triangulation (Fig.[^b)); then re-triangulate the hole left by the deleted 
tetrahedra (Fig. [^c)). New point insertion: insert the new point Pnew into the 
triangulation and add to the set U all the rays stored in the conflicting tetrahedra 
(Fig. gd -f)). Rays removal: for each tetrahedron of the entire triangulation 
remove the rays ending in Poid- Ro-y casting: cast one ray for each ray in U. 

In our case, whenever the 3D estimate of a point moves, we apply the Re¬ 
finement Event Handler, before point addition and region growing, if and only 
if the point is inside the shrinked volume Dt^. (see Section^, otherwise we do 
not move the point (this second case happens very rarely [8]). 


Table 1. Complexity analysis; 


U }} 


means not existing step. 


Step 

straightforward 

algorithm 

K 

heuristic 

proposed 

algorithm 

window 

heuristic 

Rays collection 

0{F ■ N^) 

OiN) 

- 

- 

Weight collection 

- 

- 

OiN) 

OiN) 

Vertex Removal 

0{N) 

OiN) 

OiN) 

OiN) 

New points insertion 

0{F ■ ■ N) 

OiN) 

0(F ■ N^ ■ N) 

OiN) 

Rays removal 

0{N^ ■ F ■ N^) 

OiN^) 

- 

- 

Weight Update 

- 

- 

OiN) 

OiN) 

Backward ray casting 

- 

- 

OiN^ ■ F) 

OiN) 

Ray casting 

0{N^ ■ F) 

0(1) 

0(Ar^ ■ F) 

0(1) 

Overall complexity 

0{N'^ ■ F) 

O(Af^) 

0(W^ • F) 

OiN) 


Complexity The number of rays involved in space carving algorithms is 0(F • 
iV^) where F and N represent respectively the number of frames and the num¬ 
ber of points in the triangulation nni, and the number of tetrahedra in a 3D 
triangulation grows quadratically with the number of points In Table[^ 

we reported the complexities for each of the previous stage; since our implemen¬ 
tation exploits the CGAL [T^ 3D triangulation data structure, the complexity 
of a single Ray casting, i.e., a cast of a single ray, is 0{N) in the general case, 
but we bound the size of the viewing ray, to avoid to include too far uncertain 
3D points estimates, so the final complexity becomes 0(1) (see [121 p.94]). 

From the analysis in the table is quite clear that this straightforward solution 
is not scalable, especially for the dependence between the number of rays and 
the number of processed frames. 


Forgetting Heuristic Lovi et al. m proposed a forgetting heuristic to limit 
the number of rays stored in each tetrahedron to a fixed number K, thus making 
the complexity independent from the number of the processed frames. However, 
we show in Section that, when the points are moving, the reconstruction is 
very inefficient even with this heuristic. 


3.2 The efficient approach 


Our contribution in this paper is an approach to deal with moving points, differ¬ 
ent from the straightforward variation of |10j . Indeed in our proposal, we avoid 
storing the list of rays inside each tetrahedron, and we just store the weight as¬ 
sociated with it. This allows the incremental reconstruction algorithm of Section 
2.1 and, at the same time, we are able to bound the temporal complexity. 

The main difficulty in the proposed approach is updating coherently the 
weights whenever a point moves, i.e., when the point is removed from the tri¬ 
angulation and added as a new point. As soon as the point is removed from the 
triangulation, we perform a backward ray casting with negative weights for each 
viewing camera such that the influence of the point is neglected. Then we remove 
the point, and we add a new vertex in the new position. Finally, we perform the 
ray casting from each viewing camera to the new point. 






=6.8 



Fig. 2. 2D example of moving point addition in the new position after point removal 
(a brighter region corresponds to a higher weight, i.e., higher probability to be carved). 


During both point removal and addition, we have to remove a set of connected 
tetrahedra from the triangulation and add a new one. Let R — ..., 

be the set of removed off tetrahedra and A = {A^, Z\^,..., A^\ the set of the 
new ones; their associated weights are respectively Wr = {tc^ , ..., } 


and Wa = {t 


'1 ) ‘"2 5 ■ 


, The weights Wr are known, while Wa are those 


to be computed for the new tetrahedra, 
related to those tetrahedra. 


without recasting the visibility rays 


Different approaches are possible: Mean value: wf = ^ X]fe=i Weighted 
mean: let di , be the Euclidean distances between the centroids of the f-th 


tetrahedron of A and the j-th of i?; then wf = 


d, 


Ifr; Minimum distance: 


wf = wf such that j = argmin^g;^ 

Among these, the third solution gives a non-smooth outcome and, even if 
this seems counter-intuitive, it results to be more suitable for our purposes. The 
main reason is that it preserves the discontinuity between matter and free space. 
For instance in Fig. [^a) we depict a 2D triangulation where we want to add 
a new point position; in Fig. (b), (c) and (d) we show the results of weights 
update after point addition with, respectively. Mean value. Weighted Mean value 
and Minimum distance approaches. It is clear that only Fig. [^d) preserved the 
discontinuity, while in other cases becomes hard to distinguish between matter 
(lower weights) and free space (higher weights). 


In case of very sparse data, the centroids of big tetrahedra, together with 
the associated visibility information, can be far from the newly added or moved 
points, and our update policy might lead to results far from the ideal solution, 
i.e., the straightforward approach discussed in Section [XT] Our algorithm over¬ 
comes this issue thanks to the use of the (so called) Steiner points added to 
the triangulation before the actual reconstruction is performed; this idea was 
already introduced in [8]. We add Steiner points to the Delaunay triangulation 
every 5m along each axis so that they cover all the space that can be represented. 
The use of Steiner points limits the creation of very big tetrahedra, the visibility 
information becomes always local, and the update policy avoids drifts. Indeed, 
experimental results show good accuracy on varied scenes, even when lack of 
textures induces very sparse data. 










Fig. 3. Incremental reconstruction example. From up left to bottom right: original 
frame, before point positions update, points moved in the scene (red dots) and manifold 
updated. 


Complexity The complexity of the steps of our algorithm are reported in 
Table The main difference with respect to the straightforward algorithm is 
the replacement of the Rays removal to the weight update and backward casting 
which are the key of the gaining in computational complexity. The proposed 
algorithm is thus 0{F■ N'^), so, in principle, the dependence with F still remains 
and results in a non scalable solution. 


Window Heuristic We are able to bound the complexity of our algorithm 
to 0{N‘^) thanks to the following heuristic: instead of backward casting all the 
rays connecting the moving point to all the viewing cameras, we consider only 
the most recent cameras. In this case the complexity of the ray casting becomes 
0{W ■ fV^), where W is the (constant) size of the window (in our case W = 15), 
so the final complexity is 0{N‘^). 

4 Experimental validation 

To evaluate our approach, we tested the system on four different sequences of 
the KITTI dataset [5] on a 4 Core i7-2630QM CPU at 2.2Ghz (6M Cache), 
with 6GB of DDRS SDRAM. The video stream was captured by a Point Grey 
Flea 2, which records 1392x512 gray scale images at 10 fps. The vehicle pose 
are estimated through a RTK-GPS and they are the initial input of our system 
together with the video stream. 

Among all the sequences we choose the 0095 (268 frames) and 0104 (313 
frames) since they depict two different urban scenarios: the former shows a nar¬ 
row environment where the building fagades are close to the camera, the latter 
captures a wide road. We also tested our approach on sequences 03 (801 frames) 
and 04 (271 frames) from the odometry dataset: these videos provide a var¬ 
ied landscape mixing natural (trees and bushes) and man-made (houses, cars) 
features. 

To provide a quantitative evaluation we compared the reconstructed meshes 
with the very accurate point clouds measured by the Velodyne HDL-64E sensor 



in the KITTI dataset through the CloudCompare tool [4]. This tool computes 
the reconstruction error as the average of the distances between each Velodyne 
point and the nearest triangle in the reconstructed mesh. 


We evaluated the performance and the accuracy of the Lovi’s approach 
against our three different updating policy. As explained previously, no man¬ 
ifold incremental reconstruction approach deals with moving points, so a fair 
comparison results to be between the straightforward approach of Lovi, ap¬ 
plied to manifold reconstruction (Section 3.1) and our updating policies. In 
Fig. [I we show an example of the reconstruction results before and after the 
red points has been moved in the Delaunay triangulation (see also the video at 
http://youtu. be/_-q9sK j cOCO). 

Fig. a shows the results of the comparison where we applied the window 
heuristic (Section 3.2) to all the algorithms. In the case of Lovi’s algorithm we 
applied the forgetting heuristic with K = 5 and K = 1, where K is the number 
of viewing rays stored for each tetrahedron. Fig. j^a) shows that the accuracy 
of the proposed approach, i.e, moving point management through minimum 
distance weight updates, is comparable with respect to Lovi’s proposal outcomes, 
where the algorithm with K = 5 stores more information, so it performs better. 
We compared our approach with respect to Lovi’s method instead of the other 
incremental reconstruction algorithm presented in [S]; the reasons are twofold. 
In [5] Litvinov and Lhuiller does not deal with moving points, which is the main 
point addressed in this paper. Moreover, Litvinov and Lhuiller point out in [3] 
that the ideal solution for a manifold reconstruction algorithm is represented 
by the manifold including as much as free space tetrahedra as possible. Since 
the solution provided by Lovi et al. coincides with the (non-manifold) mesh 
containing all the free space tetrahedra, a reconstruction accuracy similar to 
Lovi’s suggests that the reconstruction is near to the ideal solution. In some 
cases our algorithm reaches even better accuracy, this is due to the smoothing 
effect induced by our heuristic. 


Fig. I^a) shows that the Minimum Distance always outperforms the other 
two updating schema as expected (see the Section 3.2). 


In Fig. I^b) we report the time performance of the algorithms. Let Tmov 
and Tnon-mov be the overall processing time with and without moving points, 
and A^mov be the number of the total points moves, e.g., if one point moves 
three times, A^mov = 3. The overhead introduced in the whole reconstruction 
process for each move of each point has been computed as Thov-Tiwwqy , The 
performance of the different update schema we presented in Section j3.2| is very 
similar since the steps involved are basically the same: for each update on the 
Delaunay data structure, we iterate over the old tetrahedra to collect the weights, 
then we iterate over the new tetrahedra to set the new weights. As expected by 
Section |3.2[ our algorithm clearly outperforms Lovi’s approach. Our updating 
schema is very efficient for two reasons. First, we only need to update locally 
the visibility, while Lovi’s approach casts a ray for each visibility ray stored 
inside the tetrahedra. Second, when we remove a point (first step of moving 
point management), we perform a ray casting backward to update only the 









(a) Absolute errors (m). 
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(b) Per-point overhead (s). 


Fig. 4. Experimental evaluation of the proposed approach with respect to Lovi ’sM- 


convenient tetrahedra, instead of iterating over the whole triangulation to remove 
the visibility rays involving the point moved as in [in]. 

5 Conclusion and future work 

In this paper we have shown that manifold reconstruction from sparse data with 
moving points is not a trivial task. To keep the Delaunay property valid when a 
point moves inside the Delaunay triangulation, we have to remove it and add a 
new point in the new position. This induces the removal of a set of tetrahedra, 
with the associated visibility information; then, we have to add a new set of 
tetrahedra with coherent visibility information; finally we have to update the 
visibility information in all the tetrahedra affected by the point move. 

Existing solutions successfully applied for classic Space Carving, result to 
be inefficient and slow when applied in the manifold reconstruction setting. In 
this setting, we investigated different approaches to handle visibility information 
propagation, by updating the weight for each tetrahedron, which roughly rep¬ 
resents the number of ray intersections, and we proposed an efficient algorithm 
to conveniently update it. We tested our system with the KITTI dataset and 
it clearly outperforms the existing approach of Lovi et al. |10j for incremental 
manifold reconstruction. 

Future works would include a photometric refinement of the manifold ex¬ 
tracted incrementally, and an evaluation of the manifold quality on-the-fly, re¬ 
lying on the uncertainty information carried by the estimation of 3D points. A 
natural extension could also deal with the reconstruction of non-rigid shapes 
whose 3D points are moving. 
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